Skip to content

crash_handler: truncate a panic message that does not fit the trace string instead of dropping it - #38950

Open
robobun wants to merge 5 commits into
mainfrom
farm/8c3f78b7/crash-handler-truncate-panic-message
Open

robobun wants to merge 5 commits into
mainfrom
farm/8c3f78b7/crash-handler-truncate-panic-message

Conversation

@robobun

@robobun robobun commented Aug 15, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • A CrashReason::Panic whose message compresses to more than the room left in the trace string (about 680 bytes on a typical Linux trace, less with more or longer frames) produces a trace string that ends right after the 0 reason tag: the frames are there, the message is gone, and nothing reports it. That string is what gets printed and what report() uploads. The pinned bun.report decoder (bun-tracestrings, parsePanicMessage) throws on the empty payload, so its parse() returns null and bun.report rejects the whole report, frames included.
  • Cause: encode_trace_string (src/crash_handler/lib.rs) compressed the message into a 2048-byte buffer, base64-encoded that into another 2048-byte buffer, and appended the result with a single write_all to the 1024-byte trace_str_buf its callers render into. BoundedArray::append_slice is all-or-nothing, so once header + frames + base64 exceeded 1024 bytes the append failed as a unit, and TraceString's Display impl discarded the error (let _ = encode_trace_string(..)). The two 2048-byte intermediates only hid the real limit.
  • Observed with a debug build and a message argument on the crash_handler.panic() test hook: a 2048-character incompressible ASCII message and a 1024-character CJK message both produced a 117-byte trace string with an empty payload; 900, 1000 and 1024 characters of base64-alphabet text (all within the 1024-byte cap the Rust panic hook applies) did not fit either. An 11-character message produced a 143-byte trace string that decoded normally.

Fix

  • encode_trace_string writes into the TraceStringBuf (BoundedArray<u8, TRACE_STRING_MAX_LEN>) itself instead of an arbitrary writer, so when it reaches the message it knows how much room is left. write_panic_message turns that room (minus the /view suffix when one follows) into a maximum compressed size and compresses the longest prefix of the message whose zlib stream fits: the whole message is tried first, and only if that fails does it bisect the cut. The cut is backed up to a UTF-8 character boundary (utf8_prefix); a message that fits is passed through byte for byte.
  • Why this is correct: the prefix's stream is a complete zlib stream, so bun.report inflates it with the decoder it already has (a stream cut short would not inflate at all, since its final block and Adler-32 trailer would be missing), and the wire format does not change. A message that fits encodes exactly as before; the two "encoded whole" test cases pass on the unfixed encoder as well.
  • Why fit-to-room rather than a bigger buffer plus a message cap: the part of the trace string before the message has no fixed size. The base URL is BUN_CRASH_REPORT_URL verbatim; each of the up to 20 frames that falls outside bun's own image carries that module's file name on Windows (StackLine::from_address), so a crash with its frames inside a native addon with a 30-character file name can put several hundred bytes of names ahead of the message; crash_handler: put the executable's debug id in the trace string #38838 adds more header fields. With a few hundred bytes of header even an ordinary 1 KB unwrap() message no longer fits today, and any fixed split between buffer size and message cap can be pushed past the same way by a longer base URL or longer module names, which is how the 2048-byte intermediates came to be wrong. Computing the budget from the bytes actually left is the only shape under which the encoder cannot emit an undecodable string; it also makes TRACE_STRING_MAX_LEN a plain knob (raising it to keep more of long messages is a one-line change that is safe either way), and it covers panic_impl and Bun__crashHandler, whose messages are not capped anywhere.
  • Cost: the overflow path makes at most about log2(len) + 2 compress2 calls, each the same one-shot call (and the same internal deflateInit allocation) the current code makes once; zlib stops as soon as the output buffer (at most 768 bytes) is full, so no attempt scales with the message. Scratch space is 768 + 1024 bytes plus a 1024-byte local in the Display impl, down from 2048 + 2048.
  • TraceString's Display impl encodes into a local TraceStringBuf and forwards it, so the five existing TraceString sites (two panic paths, three ZigError stderr paths) are unchanged, and an encoding error still leaves whatever was encoded to be printed, as before. TraceStringAction loses its PartialEq derive, which the removed == was the only user of.
  • Both trace_str_buf declarations become TraceStringBuf so they cannot drift from the encoder's limit. The one in rust_panic_hook is deleted by crash_handler: report Rust panics through crash_handler() and test the panic hook #38927, which routes Rust panics through crash_handler(), so merging the two PRs needs that one line resolved. crash_handler: report Rust panics through crash_handler() and test the panic hook #38927 also caps the hook's message at 1024 bytes; as measured above that alone does not avoid this bug, and with this change any length yields a decodable report.
  • crash_handler.panic() in bun:internal-for-testing takes an optional message (src/runtime/api/crash_handler_jsc.rs). Without one it panics with the same message as before, so the existing assertions on it are unaffected.
  • Verified:
    • test/cli/run/run-crash-handler.test.ts, trace string panic message: decodes the trace string by walking the format (header, feature VLQs, frames, 0 tag) and inflating the payload with a fatal UTF-8 decoder. A short and an empty message decode whole. Two messages built from TRACE_STRING_MAX_LEN bytes of hash output (as hex, and as one 3-byte CJK character per byte) decode to a non-empty strict prefix of the message, with the trace string within 64 bytes of the limit, so the cut is the longest one that fits rather than an arbitrary shorter one; the CJK case also proves the cut lands on a character boundary. Since the payload can hold at most 3/4 of TRACE_STRING_MAX_LEN compressed bytes and that much entropy cannot compress below it, these messages cannot fit on any platform or header size, and a future change to the constant keeps them unfittable.
    • With the test hook but without the encoder change, the two long-message cases fail with panic reason has no payload: the message was dropped; with the released bun all four fail because the hook ignores its argument; with bun bd test all four pass (the hex message is cut to 1213 characters, the CJK one to 413, both trace strings exactly 1024 bytes). The whole file passes (21 pass, 9 platform skips), as does test/cli/run/crash-report-command-char.test.ts; the Windows x64 and arm64 CI lanes ran the new cases as well.
    • cargo check -p bun_crash_handler for x86_64-pc-windows-msvc, aarch64-pc-windows-msvc, aarch64-apple-darwin and x86_64-unknown-linux-gnu (uLong is 32-bit on Windows; compress_message treats a length that does not fit it as a prefix that does not fit), and cargo clippy on the crate is clean.

Background

  • Trace string: the https://bun.report/<version>/<payload> URL the crash handler prints and optionally uploads. The payload is positional: platform and command characters, a format version character, 7 characters of git sha, then VLQ-encoded fields (feature bits, one entry per stack frame, a 0 terminator), then a reason character and its payload. A frame outside bun's own image is encoded as the module name plus an offset, which is the variable-size part of the header. For a panic the reason is 0 and the payload is the zlib stream of the message in base64 with the padding stripped; bun.report's decoder inflates everything after the tag. The string is assembled in a stack buffer in the crash handler, which is where the 1024-byte limit comes from.
  • VLQ: the base64 variable-length integer encoding that source maps use; every numeric field in the payload is one.
  • BoundedArray<u8, N>: a fixed-capacity array plus a length, used so the crash path does not allocate. Its write_all appends the whole slice or nothing.
  • Unpadded base64 takes ceil(4n / 3) bytes for n input bytes, which is where budget * 3 / 4, the largest n whose encoding fits in budget bytes, comes from.
  • compress2 is zlib's one-shot compressor. When the output buffer is too small it returns Z_BUF_ERROR as soon as the buffer is full instead of compressing the rest of the input, which is what keeps the retries cheap.

…tring instead of dropping it

encode_trace_string wrote the whole base64 zlib stream of the panic
message in one append to the 1024-byte trace string buffer. When the
header, frames and stream together exceeded the buffer, the append
failed as a unit, the error was swallowed by TraceString's Display impl,
and the report went out with a '0' reason and no payload, which
bun.report cannot parse.

The encoder now writes into the bounded buffer itself, so it knows how
much room is left, and compresses the longest prefix of the message
whose stream fits there (found by binary search, cut on a UTF-8
boundary), leaving space for the /view suffix when one follows.

The crash_handler.panic() test hook takes an optional message so the
test can drive this.
@coderabbitai

coderabbitai Bot commented Aug 15, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@robobun, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 32 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 04e426e2-01c8-4a11-9b62-7cd4005cb364

📥 Commits

Reviewing files that changed from the base of the PR and between 9fb606f and 1cf7ec0.

📒 Files selected for processing (4)
  • src/crash_handler/lib.rs
  • src/js/internal-for-testing.ts
  • src/runtime/api/crash_handler_jsc.rs
  • test/cli/run/run-crash-handler.test.ts

Comment @coderabbitai help to get the list of available commands.

@robobun

robobun commented Aug 15, 2026 •

Copy link
Copy Markdown
Collaborator Author

Status: ready for a maintainer.

Reproduced on a debug build by giving the crash_handler.panic() test hook a message: a 2048-character incompressible message (and a 1024-character CJK one) produced a 117-byte trace string ending in the bare 0 tag, which the pinned bun.report parser rejects outright. With this change the same crashes produce a 1024-byte trace string whose payload inflates to a prefix of the message (1213 and 413 characters respectively).

Test: trace string panic message in test/cli/run/run-crash-handler.test.ts. Its two long-message cases fail on the unfixed encoder with panic reason has no payload: the message was dropped and pass with the fix; the rest of the file and crash-report-command-char.test.ts still pass.

CI: every lane that has run is green on the current head (177 of 179 jobs in build 98095, including Windows x64 and arm64 and the Linux ASAN lanes; the remaining entries flagged in the build are unrelated tests that passed on retry). The only job not accounted for is darwin 14 aarch64 - test-bun, which has expired waiting for an agent in both builds of this PR; the change is platform-neutral and the crate also passes cargo check for the darwin targets. I am not going to keep re-triggering for that lane.

While probing this I found a separate problem, tracked separately: the crash handler's raw stderr writer ignores EAGAIN, so a crash while stderr is a non-blocking pipe or socket that is momentarily full loses part of the report. The test here keeps the child's argv short to stay clear of it.

@github-actions

Copy link
Copy Markdown
Contributor

This PR may be a duplicate of:

  1. crash_handler: report Rust panics through crash_handler() and test the panic hook #38927 - Also fixes an over-long panic message being reported as an empty trace-string payload (caps it via msg.floor_char_boundary(1024) citing encode_trace_string), touches the same four files, and adds its own inflateSync-based payload decoder to run-crash-handler.test.ts.

🤖 Generated with Claude Code

@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

Not a duplicate of #38927, though the two touch the same files and should both land.

#38927 changes how Rust panic! payloads reach crash_handler() and caps that one caller's message at 1024 bytes. It does not change encode_trace_string, which is where the message is lost: the trace string has about 910 bytes of room after the header and frames on a typical Linux trace (less on Windows, where frames carry DLL names), and anything whose compressed form does not fit is dropped whole. Messages within #38927's cap still hit that. Measured on this branch's build, which cuts instead of dropping: base64-alphabet text of 900, 1000 and 1024 characters is cut to 857 characters in each case, so each of those would come out of the unfixed encoder as an empty payload. The cap also only applies to the panic hook; panic_impl and Bun__crashHandler messages are not bounded at all.

This PR fixes the encoder, so every CrashReason::Panic producer gets a decodable report whatever the message length. The overlap with #38927 is one line (it deletes the rust_panic_hook buffer this PR retypes) plus two independent test helpers in run-crash-handler.test.ts; whichever lands second can drop one of the helpers.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. The binary-search truncation logic in write_panic_message and its interaction with utf8_prefix are non-trivial enough that a human look would still be worthwhile.

Checked: binary-search termination and the too_long == fits exit (only reachable when even the empty stream overflows the budget); utf8_prefix monotonicity so bisection stays well-ordered; that TRACE_STRING_MAX_LEN - writer.len() cannot underflow; that the OpenIssue path the test exercises has an empty suffix so the decoder's payload.slice(i+1) is exactly the base64; and that encode_len_from_size is const fn so B64_LEN compiles.

Extended reasoning...

Overview

This PR fixes a bug where a panic message that compressed to more than the room left in the 1024-byte trace-string buffer was silently dropped, leaving a 0 reason tag with no payload — which bun.report's decoder rejects, discarding the whole report. The fix changes encode_trace_string to take the concrete TraceStringBuf so it can measure remaining capacity, then binary-searches for the longest UTF-8-aligned prefix of the message whose zlib stream fits in the remaining space (less any /view suffix). Four files touched: src/crash_handler/lib.rs (encoder), src/runtime/api/crash_handler_jsc.rs and src/js/internal-for-testing.ts (test hook now accepts an optional message), and test/cli/run/run-crash-handler.test.ts (four new subprocess tests that decode the trace string end-to-end).

Security risks

None. This is diagnostic-output formatting on the crash path. No new inputs cross a trust boundary; the panic message was already being fed to zlib and base64. The only new unsafe block is the existing compress2 FFI call moved into a helper with the same bounds.

Level of scrutiny

Medium. The crash handler is not on any request-serving path, and the failure mode of a bug here is a malformed or over-truncated trace string while the process is already dying — no worse than the status quo, which drops the message entirely. That said, trace strings are how production crashes get reported, so a regression that broke them for common (short) messages would hurt debuggability. The two "encoded whole" tests guard that path, and the PR description confirms they pass on the unfixed encoder too.

Other factors

I traced the binary search's invariants: fits is only ever assigned a length whose (possibly backed-up) prefix has been proven to compress within budget, too_long is only assigned a length proven not to; len = fits + (too_long - fits)/2 keeps len in [fits, too_long), so the interval shrinks each iteration and too_long - fits cannot underflow. The break condition too_long - fits == 1 fires with a fresh compressed_len in hand. The too_long == fits early error is reachable only when even the empty zlib stream exceeds the budget (i.e. essentially zero room left), which is the same degenerate case the old code could not handle either. utf8_prefix is non-decreasing in len, so bisection stays well-ordered even though it may back up a few bytes; zlib non-monotonicity can only make the chosen prefix slightly sub-optimal, not incorrect. I also confirmed encode_trace_string's only caller is the Display impl (the two trace_str_buf sites go through it via write!), that the removed PartialEq/Eq derives are dead after the == became a match, and that the test's decoder correctly assumes no /view suffix because --debug-crash-handler-use-trace-string forces the OpenIssue action.

Deferring rather than approving because ~90 lines of new algorithmic Rust in the crash handler — a binary search over compressed sizes with a UTF-8 boundary adjustment — is past the threshold I'd auto-approve without a human sanity check, even though I found nothing wrong with it.

@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

For whoever takes the human look the review above asks for, the parts worth reading closely are write_panic_message and utf8_prefix in src/crash_handler/lib.rs, about 40 lines together. Two notes on coverage:

  • The reserved argument (room kept for /view) is only ever non-zero for a panic on the debug-build path that symbolizes the trace locally, and that path builds the trace string only to hand it to report(), which is disabled there, so no test can observe it. The three ViewTrace sites that do print go through ZigError reasons, and the existing tests in the file (segfault, abort, trap, out of memory, plus the command-char file) cover the non-panic reasons through the rewritten encoder.
  • Both "encoded whole" cases pass on the current encoder as well; they are there to pin the common path. The two long-message cases are the ones that fail without the change.

Comment thread src/crash_handler/lib.rs Resolved
Comment thread src/crash_handler/lib.rs Resolved
Comment thread src/crash_handler/lib.rs Resolved
Comment thread src/crash_handler/lib.rs Resolved
Comment thread src/crash_handler/lib.rs Resolved
Comment thread src/crash_handler/lib.rs Resolved
Comment thread src/crash_handler/lib.rs Resolved
Comment thread src/runtime/api/crash_handler_jsc.rs Resolved
Comment thread src/crash_handler/lib.rs Resolved
Comment thread src/crash_handler/lib.rs Resolved
@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

The comments the comment linter flagged are cut down to one line each in ca5ba6e and 49df563 (the SAFETY: one stays, since undocumented_unsafe_blocks is denied for the workspace). No code changes in either commit; the four new tests still pass on the debug build.

@robobun

robobun commented Aug 15, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 12:15 PM PT - Aug 15th, 2026

❌ @robobun, your commit 1cf7ec0 has some failures in Build #98095 (All Failures)


🧪   To try this PR locally:

bunx bun-pr 38950

That installs a local version of the PR into your bun-38950 executable, so you can run:

bun-38950 --bun

@robobun

robobun commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator Author

Self-review raised one design question worth recording: why fit the message to the room left rather than just enlarge the buffer and cap the message at the producers (every in-tree producer is at or under 1024 bytes already). The PR body now has a bullet on that under Fix; the short version is that the header is not fixed size (the base URL comes from BUN_CRASH_REPORT_URL, and on Windows every frame outside bun's image carries its module's file name), so a fixed split can always be pushed past, while computing the budget from the bytes actually left cannot emit an undecodable string and leaves the buffer size as a free knob.

1cf7ec0 also sizes the test's unfittable messages off TRACE_STRING_MAX_LEN, so they stay unfittable if that constant is ever raised.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. Because it adds ~90 lines of new logic (a binary search over zlib compression attempts, plus an unsafe FFI call) to the crash handler — code that must not itself misbehave while the process is already dying — a human look at write_panic_message and utf8_prefix would still be worthwhile, as the author already noted.

What was reviewed:

  • Binary-search termination in write_panic_message: fits < too_long shrinks each iteration, the too_long == fits guard catches the nothing-fits case, and the final dest[..compressed_len] always holds the last successful compression.
  • Base64 budget math: floor(3·budget/4) compressed bytes → unpadded base64 ≤ budget, and B64_LEN = encode_len_from_size(768) = 1024 is large enough for the padded intermediate; reserved bytes are left for the suffix.
  • utf8_prefix: Rust's & binds tighter than ==, so message[len] & 0xC0 == 0x80 parses as intended; the len < message.len() guard passes the whole message through untouched.
  • TraceStringAction losing PartialEq/Eq: no remaining == comparisons on it; all callers still go through the Display impl, whose local 1024-byte buffer copies into an empty caller buffer of the same size.
Extended reasoning...

Overview

The PR fixes a bug where a panic message that compresses to more than the room left in the 1024-byte trace-string buffer was silently dropped (the all-or-nothing BoundedArray::append_slice failed and the error was discarded), leaving a bare 0 reason tag that bun.report's decoder rejects. The fix changes encode_trace_string to write into the concrete TraceStringBuf so it knows how much room remains, and adds write_panic_message, which binary-searches for the longest UTF-8-boundary prefix of the message whose zlib stream fits that room. It also introduces a TraceStringBuf type alias and TRACE_STRING_MAX_LEN constant so the two callers can't drift, extends the crash_handler.panic() test hook to accept a message, and adds four subprocess tests that decode the trace string end-to-end (walk header/VLQs/frames, inflate the payload with a fatal UTF-8 decoder).

Security risks

None identified. The input is a panic message that Bun itself produced; compress2 is called with correct pointer/length pairs (verified against the slice bounds); the Windows uLong narrowing is handled by treating a length that doesn't fit as "prefix doesn't fit". No user-controlled data reaches a new trust boundary.

Level of scrutiny

High. The crash handler runs after the process is already in a bad state, must not allocate, and must not itself crash — a bug here loses the very diagnostics it exists to preserve. The new code adds ~2.8 KB of stack locals (768 + 1024 in write_panic_message plus 1024 in the Display impl), an unsafe zlib FFI call in a loop, and a bisection whose termination depends on invariants I traced by hand rather than by test. That's beyond "mechanical", even though I believe it is correct.

Other factors

  • The tests are well-constructed: entropy-derived messages guarantee incompressibility on every platform, the CJK case proves the cut lands on a char boundary via a fatal TextDecoder, and the "encoded whole" cases pin the common path. The author verified the long-message cases fail on the unfixed encoder.
  • The comment-cop bot's flags were addressed (comments cut to one line each; all threads resolved).
  • There is a known one-line merge overlap with #38927 (which deletes the rust_panic_hook buffer this PR retypes) that whoever lands second will need to resolve.
  • The Display impl now double-buffers (encode into a local, then copy to the caller's buffer). Both are 1024 bytes and the caller's starts empty at every call site, so the copy always succeeds and behavior is preserved for non-panic reasons.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant